Collective Communication in Wormhole-Routed Massively Parallel Computers

Massively parallel computers (MPC) are characterized by the distribution of memory among an ensemble of nodes. Since memory is physically distributed, MPC nodes communicate by sending data through a network. In order to program an MPC, the user may directly invoke low-level message passing primitives, may use a higher-level communications library, or may write the program in a data parallel language and rely on the compiler to translate language constructs into communication operations. Whichever method is used, the performance of communication operations directly aaects the total computation time of the parallel application. Communication operations may be either point-to-point, which involves a single source and a single destination, or collective, in which more than two processes participate. This paper discusses the design of collective communication operations for current systems that use the wormhole routing switching strategy, in which messages are divided into small pieces and pipelined through the network. Compared to the store-and-forward switching method that was used in early multicomputers, wormhole routing often reduces the eeect of path length on communication latency. Over the past several years, a number of researchers have exploited this property in the design of new collective communication algorithms, which diier fundamentally from their store-and-forward predecessors. This paper discusses the signiicant issues involved in wormhole-routed collective communication and presents the major classes of solutions that have been proposed to address the problem.